When Nighthawk makes an incorrect classification, submitting feedback will help improve the model’s performance in the future. This document outlines the steps for submitting feedback on Nighthawk annotations using Raven Pro sound analysis software.
Jump to FAQ (bottom of page).
IMPORTANT: Be sure to save your selection table!
--raven-output.Run Nighthawk across a file and output a Raven selection table with
the --raven-output flag. Here, we do this on the test3.wav
file included in this repository.
nighthawk test_inputs/test3.wav --raven-output
This command creates the output files
test_inputs/test3_detections.txt and
test_inputs/test3_detections.csv.
In Raven Pro, use File -> Open Sound Files and choose the input
audio file (in this case, test3.wav). Open a spectrogram
view with your favored parameters. Then, use File -> Open Selection
Table and choose the corresponding .txt output (in this case,
test3_detections.txt). Your Raven window will now look
something like this:
Create additional annotation columns in the selection table for feedback. Right-click on the selection table header and select “Add Annotation Column”:
Name the new annotation column category_update:
You have now added an empty category_update column to
the selection table:
category_update
column.There are four types of feedback you can provide: (1) specifying false positives; (2) correcting any incorrect identifications; (3) confirming Nighthawk’s identifications; and (4) adding missing detections. Any of these are helpful - you can provide one, two, or all types of feedback. For example, you can provide feedback on only false positives (and not confirm, correct, or add anything).
nIf any Nighthawk detections actually contain no flight calls, enter
n in the category_update column to specify
that there are no flight calls in the selection. Use
n for any of the following cases:
IMPORTANT: There must be NO flight
calls present in the selection to use n. DO
NOT use n if there are any flight calls present in
the selection.
Be sure to save your selection table with any updates!
The selection in question:
Nighthawk thinks there is a Parulidae (warbler) call here, but we
don’t see an obvious call. Enter n in the
category_update column to specify that there are
no flight calls in this selection:
For Nighthawk detections that are identified incorrectly (or could be
identified more accurately) in the predicted_category
column, do the following:
Enter the correct classification in the
category_update column. The category must be one of the
following:
code column in nighthawk/taxonomy/ebird_taxonomy.csv)group column in nighthawk/taxonomy/groups_ebird_codes.csv)family column
in nighthawk/taxonomy/ebird_taxonomy.csv)order column in nighthawk/taxonomy/ebird_taxonomy.csv)Shrink the selection to cover only the identified call.
Be sure to save your selection table with any updates!
The selection in question:
Nighthawk thinks there is a Passeriformes (passerine) call here. We
think it can be confidently identified to family Turdidae (thrushes), so
we enter Turdidae in the category_update
column:
We make sure to shrink the selection so it defines a box around the specific call in question:
c and shrinking the
selectionFor Nighthawk detections that are identified correctly (in the
class column), do the following steps:
c in the category_update
column.The selection in question:
We think this is indeed swathr (Swainson’s Thrush), so we enter
c in the category_update column:
We make sure to shrink the selection so it defines a box around the specific call in question:
If you encounter flight calls that were not detected by Nighthawk,
you can create a new selection and add the correct classification to the
category_update column:
category_update column
corresponding to the new selection.IMPORTANT: DO NOT add selections for vocalizations that are not flight calls.
Note that for this example, which uses test3.wav, I have
deleted some of Nighthawk’s detections in order to demonstrate how to
add new detections. If you run Nighthawk on test3.wav, you
will likely not find any missed calls.
The audio in question. We see a faint Parulidae (warbler) call that Nighthawk has not detected:
We draw a new selection box around the call. The box borders are dashed, which means we have not yet committed the selection:
After committing the selection, the box borders are solid:
We add our ID to the category_update column:
prep_nighthawk to compile feedback files.Once you are finished reviewing a file, make sure to save the
selection table (.txt file). Then, create a new YAML file with metadata
about the recording. A YAML file is simply a text file with a particular
formatting. Use test3.yml as a template. The contents of
that file are shown here for easy reference:
recordist:
name: Benjamin Van Doren
email: vandoren@cornell.edu
location:
name: Ithaca_House
latitude: 42.44
longitude: -76.50
equipment:
microphone: Sennheiser XS Lav Mic
recorder: iPhone 8
accessories: Belkin charge adapter and Anker powerbank
recording_session:
start_time_utc: 2023-05-11 01:16:00
You should now have three different files:
An audio file that was run through Nighthawk
(e.g. test3.wav)
A selection table (txt) file edited in Raven
(e.g. test3_detections.txt)
A YAML file with metadata about your recording
(e.g. test3.yml)
The final step is to run a utility that comes with the Nighthawk
package, called prep_feedback. This command-line program
takes audio, txt, and yml files and outputs a single .tar.gz archive to
upload to the Nighthawk developers. For example, this is how we would
run prep_feedback on our example files:
prep_feedback --audio test3.wav --txt test3_detections.txt --yaml test3.yml
This outputs the following:
NOTE: Please ensure that the recording start time entered
in your YAML file is in Universal Coordinated Time (UTC).
Checks passed.
Writing archive BENJAMINVANDOREN_ITHACA-HOUSE_42.44_-76.50_20230511_011600_Z.tar.gz.
Done. Please send this file to Nighthawk developers.
The utility has now created a new file, a .tar.gz archive called
BENJAMINVANDOREN_ITHACA-HOUSE_42.44_-76.50_20230511_011600_Z.tar.gz.
Now, please send us your .tar.gz file. Contact Benjamin Van Doren (vandoren@cornell.edu) for
instructions on submitting files, or for questions about errors
encountered when running prep_feedback. This program runs
various checks on the files to make sure they are consistent with our
expectations; missing columns or typos of species, families, or orders
will case errors.
No. Only entries in the category_update column will be
incorporated into further model training. Any unreviewed detections will
have blank entries in category_update and will be
ignored.
No problem! Feel free to only enter n in the
category_update column where Nighthawk has returned a false
positive, and leave everything else blank. All other unreviewed
detections will have blank entries in category_update and
will be ignored.
No. Vocalizations that are not flight calls should be ignored (or
marked as n if they triggered a Nighthawk detection).